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itle of the Invention] 
Storage System 
[Abstract] 
[Object] 

An object of the. present invention is to, in remote 
copies made among storage controllers that are located in N 
sites (the number of sites is in general three or more) , 
manage differences that are used to quickly make remote-copy 
data coincide with one another among the remaining sites 
after an arbitrary site suffers from disaster. 
[Solving Means] 

An update history of a remote-copy range in a site, 
which is directly updated from a host computer, and an 
update history of a remote-copy range in another site, which 
is a copy destination of the remote-copy range in question, 
are shared between the sites in question. Writing of the 
update histories is triggered by a certain update from the 
host computer. When an arbitrary site suffers from disaster, 
contents of the remote-copy range are made coincide by 
copying, according to the update history, part of contents 
of the remote-copy range existing between arbitrary sites 
that do not suffer from disaster. 
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[Claims] 
[Claim 1] 

A storage system characterized in that: 
in a remote-copy configuration so formed that storage 
systems are connected to one another through a network among 
sites, said sites each including a host computer and a 
storage system that is connected to the host computer, and 
data updated by the host computer is reflected for a storage 
system in another site, 

an update history of a remote-copy specified range of 
a storage system, which is directly updated by the host 
computer, and an update history of a copy-to remote-copy 
specified range of a storage system in another site, which 
is a remote-copy destination of the remote-copy specified 
range, are shared between storage systems each including a 
remote-copy specified range and a copy-to remote-copy 
specified range, writing of the update histories being 
triggered by a certain update from the host computer, and 

when an arbitrary site suffers from disaster, by 
copying part of contents of a remote-copy specified range or 
part of contents of a copy-to remote-copy specified range, 
which exist between arbitrary sites that do not suffer from 
disaster, according to the update history, the contents of 
the remote-copy specified range or the contents of the copy- 
to remote-copy specified range are made to coincide with 
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each other. 
[Claim 2] 

A storage system according to Claim 1, wherein: 

areas to which the update history of the remote-copy 
specified range is written are provided in a multiplexed 
manner, and when the amount of updates from the host 
computer to the remote-copy specified range of the storage 
system, which is directly updated by the host computer, 
exceeds a threshold value, an update from the host computer, 
which triggers switching to a new update history, is 
determined, said determination being shared between storage 
systems each including a remote-copy specified range and a 
copy-to remote-copy specified range; 

in each storage system, from a point of time at which 
updates to the remote-copy specified range or to the copy-to 
remote-copy specified range reach an update from the host 
computer, which triggers switching to a new update history, 
writing to the new update history is started; and 

after checking that all storage systems, each of 
which includes a remote-copy specified range and a copy-to 
remote-copy specified range, have started writing of the new 
update history, writing of an existing update history is 
interrupted, and then the existing update history is 
initialized. 
[Claim 3] 
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A storage system according to Claim 2, wherein: 
said determination of the update from the host 
computer, which triggers switching to the new update history, 
is made also when a length of time elapsed after the last 
switching of the update history exceeds a threshold value. 
[Claim 4] 

A storage system according to Claim 2, wherein: 
exchanges of information between the storage systems 
are made only with a direct copy source, and a direct copy 
destination, corresponding to the remote-copy specified 
range, or the copy-to remote-copy specified range, of the 
storage system in question. 
[Claim 5] 

A storage system according to Claim 2, wherein: 
exchanges of information between the storage systems 
are made with said information being attached to remote-copy 
data to be transferred or received. 
[Claim 6] 

A storage system according to Claim 2, wherein: 
when an arbitrary site suffers from disaster, an 
update history to be used to make contents of a remote-copy 
specified range or contents of a copy-to remote-copy 
specified range coincide between arbitrary sites that do not 
suffer from disaster is selected in both of the sites on the 
conditions that the update history is valid, and that 



updates from the host computer which has started writing are 
the same; and 

if there are a plurality of update histories each 
satisfying the conditions in both of the sites, an update 
history whose update from the host computer which has 
started the writing is the latest is selected. 
[Claim 7] 

A storage system according to Claim 2, wherein: 
if it is detected that an arbitrary site has suffered 
from disaster, writing of an existing update history is 
continued, and at a point of time when the site which has 
suffered from the disaster is completely recovered to a 
normal state, the writing of the existing update history is 
interrupted, and then the existing update history is 
initialized . 
[Claim 8] 

A storage system according to Claim 2, wherein: 
if it is detected that an arbitrary site has suffered 
from disaster, by use of areas for update histories from 
which an existing update history is excluded, the steps 
described in Claim 2 are executed between sites that do not 
suffer from disaster. 

[Detailed Description of the Invention] 
[0001] 

[Technical Field to which the Invention Pertains] 



The present invention relates to remote-copy 
functions used to duplex data between storage systems with a 
host computer not being involved. It relates more 
particularly to, in remote copies made among respective 
storage systems located in three or more sites, when an 
arbitrary' site suffers from disaster, if the remote copying 
is continued among sites that do not suffer from the 
disaster, how to manage contents of a remote-copy specified 
range so that the contents are made to coincide with each 
other among the storage systems. 
[0002] 
[Prior Art] 

In order to prevent data in a storage system from 
being lost due to disaster or the like, the data Is duplexed 
in another storage system located in a remote place. 
Examples of this duplexing function include remote copy 
technologies . 
[0003] 

In the remote copy technologies, a target volume to 
be remote-copied is set in a primary storage controller 
connected to a host computer, and also in a secondary 
storage controller connected to the primary storage 
controller; and copies are continuously being executed so 
that contents of a primary volume of a primary storage 
device connected to the primary storage controller is always 



7 



made coincide with contents of a secondary volume of a 
secondary storage device connected to the secondary storage 
controller . 
[0004] 

The technology for writing data in duplication 
between different storage controllers with a host computer 
not being involved is disclosed in Japanese Patent Laid-open 
No. 11-85408. According to the above invention, a primary 
storage controller, which has received from a host computer 
write data to which write time is added, notifies the host 
computer that writing of the write data has been completed, 
and then transfers the write time and the write data to a 
secondary storage controller in the order of the write time. 
In a secondary storage controller, the write time and the 
write data which have been received from the primary storage 
controller are stored in a nonvolatile cache memory to 
ensure that data stored before this specified write time is 
reliable . 
[0005] 

The remote-copy operation is roughly classified into 
two: a synchronous remote copy for transferring write data 
to the secondary storage controller before notifying the 
host computer of the completion of a write request; an 
asynchronous remote copy for transferring write data to the 
secondary storage controller in asynchroni zat ion with the 
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write request after notifying the host computer of the 
completion of the write request. If a remote-copy function 
is used for applications such as a database, it is necessary 
to update a secondary volume in the same order as the update 
order in which a primary volume has been updated so as to 
ensure the update order of a log. In order to achieve the 
above, in the case of the asynchronous remote copy, every 
time a write is made from a host computer, a sequence number 
is added to write data in a primary storage controller. 
Then, the write is reflected in a secondary volume according 
to the sequence number in a secondary storage controller. 
[0006] 

In the conventional remote-copy operation described 
above, the remote-copy operation may be temporarily 
interrupted (for example, in a case where the remote-copy 
operation is interrupt by a link failure between the primary 
storage controller and the secondary storage controller, or 
the like) . In such temporal interruption, with the 
objective of making contents of the secondary volume 
coincide with those of the primary volume as quickly as 
possible when the remote-copy operation is restarted, an 
update history may be written in the primary storage 
controller as difference information between the primary and 
secondary volumes. In the* update history, updates from the 
host computer to the primary volume performed during the 
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interruption of the remote-copy operation are recorded. 
After that, when the remote-copy operation is restarted, by 
using the written update history to reflect the contents of 
the primary volume in the secondary volume, it is possible 
to make the contents of the secondary volume coincide with 
those of the primary volume. 
[0007] 

The difference information in the conventional remote 
copy usually means update locations from the host computer. 
More specifically, a location at which the primary volume is 
updated but the secondary volume is not updated is written. 
Therefore, the difference information is in general stored 
on the primary storage controller side. 
[0008] 

In recent years, with the increase in speed of 
networks, and with the decrease in cost of networks, costs 
of data transmission between remote locations are decreasing. 
For this reason, to increase the amount of transferred data 
in the remote-copy operation, and thereby to further 
increase the availability of remote-copy data, there is a 
demand that remote copies are made among N sites (the number 
of sites is three or more) so that data is kept duplexed 
even if one site suffers from disaster. 
[0009] 

As operational management of remote copies among N 



sites, if one site among N sites suffers from disaster, it 
is desirable that a remote-copy configuration be made again 
by use of the remaining (N - 1) sites so as to continue the 
remote-copy operation. In this case, in order to continue 
the remote-copy operation, it is necessary to quickly make 
remote-copy data coincide between the remaining (N - 1) 
sites. In other words, it is desirable to perform 
difference management among (N - 1) sites so that the 
remote-copy data is quickly made coincide with one another 
by copying the differences. 
[0010] 

It is because if the difference management is not 
performed, there may arise a case where it becomes necessary 
to copy all remote-copy data from a copy source to a copy 
destination. In comparison with the copy of the differences, 
the copy of all remote-copy data takes a long time, and the 
availability of data is decreased while the copy is being 
executed. Therefore, it is not possible to make full use of 
the redundancy achieved by the number of sites. 
[0011] 

Here, as an example, on the assumptions that a 
remote-copy configuration is made with three sites, and that 
the conventional difference management method is applied to 
remote-copy operation, problems which arise are considered. 
Here, three sites are designated as A, B, C. 
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[0012] 

First of all, if a flow of remote-copy data to be 
transferred among the sites are A — > B —> C, in the event 
that the site B suffers from disaster, it is desirable that 
differences be copied between a volume in the site A and a 
volume in the site C before restarting the remote-copy 
operation. Because the difference information in the 
conventional remote copy is in general stored on the primary 
storage controller side, difference management information 
between the site A and the site B is stored in the site A, 
and difference management information between the site B and 
the site C is stored in the site B. For this reason, if the 
site B suffers from disaster, the difference management 
information between the site B and the site C is lost. 
Accordingly, differences between the site A and the site C 
cannot be known. 
[0013] 

In addition, if a flow of remote-copy data to be 
transferred among the sites is A — > B and A — > C, in the 
event that the site A suffers from disaster, it is desirable 
that differences be copied between a volume in the site B 
and a volume in the site C before restarting the remote-copy 
operation. In this case, because difference management 
information between the site A and the site B and that 
between the site A and the site C are stored in the site A 
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that has suffered from the disaster, differences between the 

site B and the site C cannot be known. 

[0014] 

In order to cope with this problem, a method is 
conceivable in which difference management is performed 
beforehand between sites between which remote-copy data is 
not exchanged during the normal operation. In the above 
example, a difference between the site A and the site C is 
managed before the site B suffers from disaster; and a 
difference between the site B and the site C is managed 
before the site A suffers from disaster. However, if this 
method is used, the amount of difference information between 
sites which must be managed beforehand increases in 
proportion to the number of sites. Furthermore, it is not 
possible to manage differences in the event of unexpected 
circumstances including, for example, a case where two sites 
concurrently suffer from disaster. 
[0015] 

[Problems to be Solved by the Invention] 

The conventional difference management method used 
for remote copies between two sites does not take difference 
management among N sites into consideration. 

[0016] 

Therefore, when remote copies are being made among N 
sites, if one site suffers from disaster, and consequently, 
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the remote-copy operation is continued among the remaining 
(N - 1) sites that do not suffer from disaster, there arises 

a problem in that a remote-copy configuration can be made 

again only after transferring all remote-copy data from a 

new copy-from site to a new copy-to site. 
[0017] 

In addition, if the conventional difference 
management method used for remote copies between two sites 
is applied beforehand to a pair of' new sites when a remote- 
copy configuration is made again after a site has suffered 
from disaster, there arises a problem in that the amount of 
difference management information of each site increases 
with the increase in the number of sites. Further, if a 
plurality of sites suffer from disaster, there arises 
another problem in that differences cannot be managed. 
[0018] 

An object of the present invention is to, in remote 
copies made among storage systems that are located in N 
sites (the number of sites is in general three or more), 
after an arbitrary site suffers from disaster, manage 
differences that are used to quickly make remote-copy data 
coincide with each other among the remaining sites. Another 
object of the present invention is to keep the amount of 
information used to manage the differences constant without 
depending on the number of sites N among which remote copies 



are made. 
[0019] 

[Means for Solving the Problems] 

In order to achieve the above-mentioned objects, 
according to one aspect of the present invention, there is 
provided a storage system, wherein: 

in a remote-copy configuration so formed that storage 
systems are connected to one another through a network among 
sites, said sites each including a host computer and a 
storage system that is connected to the host computer, and 
data updated by the host computer is reflected for a storage 
system in another site, 

an update history of a remote-copy specified range of 
a storage system, which is directly updated by the host 
computer, and an update history of a copy-to remote-copy 
specified range of a storage system in another site, which 
is a remote-copy destination of the remote-copy specified 
range, are shared between storage systems each including a 
remote-copy specified range and a copy-to remote-copy 
specified range, writing of the update histories being 
triggered by a certain update from the host computer, and 

when an arbitrary site suffers from disaster, by 
copying part of contents of a remote-copy specified range or 
part of contents of a copy-to remote-copy specified range, 
which exist between arbitrary sites that do not suffer from 



disaster, according to the update history, the contents of 
the remote-copy specified range or the contents of the copy- 
to remote-copy specified range are made to coincide with 
each other. 
[0020] 

In addition, areas to which the update history of the 
remote-copy specified range is written are provided in a 
multiplexed manner, and when the amount of updates from the 
host computer to the remote-copy specified range of the 
storage system, which is directly updated by the host 
computer, exceeds a threshold value, an update from the host 
computer, which triggers switching to a new update history, 
is determined, the determination being shared between 
storage systems each including a remote-copy specified range 
and a copy-to remote-copy specified range; in each storage 
system, from a point of time at which updates to the remote- 
copy specified range or to the copy-to remote-copy specified 
range reach an update from the host computer, which triggers 
switching to a new update history, writing to the new update 
history is started; and after checking that all storage 
systems, each of which includes a remote-copy specified 
range and a copy-to remote-copy specified range, have 
started writing of the new update history, writing of an 
existing update history is interrupted, and then the 
existing update history is initialized. 



[0021] 

In addition, the determination of the update from the 
host computer, which triggers switching to the new update 
history, may be made also when a length of time elapsed 
after the last switching of the update history exceeds a 
threshold value. 
[0022] 

In addition, exchanges of information between the 
storage systems may also be made only with a direct copy 
source, and a direct copy destination, corresponding to the 
remote-copy specified range, or the copy-to remote-copy 
specified range, of the storage system in question. 
[0023] 

Further, the exchanges of information between the 
storage systems may also be made with said information being 
attached to remote-copy data to be transferred or received. 
[0024] 

Moreover, when an arbitrary site suffers from 
disaster, an update history to be used to make contents of a 
remote-copy specified range or contents of a copy-to remote- 
copy specified range coincide between sites that do not 
suffer from disaster is selected in both of the sites on the 
conditions that the update history is valid, and that 
updates from the host computer which has started writing are 
the same; and additionally, if there are a plurality of 
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update histories each satisfying the conditions in both of 
the sites, an update history whose update from the host 
computer which has started the writing is the latest may 
also be selected. 
[0025] 

Furthermore, an instruction which is used in the 
event of a failure in a remote-copy destination is given 
from a maintenance terminal, the instruction instructing to 
allow or disallow the execution of a remote copy to the 
other remote-copy destinations for each remote-copy 
destination of' a remote-copy specified range or of a copy-to 
remote-copy specified range; and when a failure occurs in a 
remote-copy destination, remote copies from the storage 
system in question to the remote-copy destinations may be 
controlled according to the instruction. 
[0026] 

[Modes for Carrying out the Invention] 

A first embodiment of the present invention will be 
described with reference to drawings below. 
[0027] 

Fig. 1 is a diagram illustrating how remote copies 
among N sites are made. In each site, each of a plurality 
of host computers 1, each of a plurality of storage systems 
2, and the like, are connected to one another through each 
local network T2 such as a LAN (Local Area Network) or a SAN 
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(Storage Area Network) . On the other hand, among the 
storage systems 2 in those sites, storage systems 2 between 
which a remote copy is made are connected to each other 
through a global network Tl. In general, the global network 
Tl is a public telephone service, which is often leased from 
a communication service provider with charge. However, the 
present invention is not limited by configurations of the 
local networks T2, and a configuration of the global network 
Tl. 
[0028] 

Target volumes to be remote-copied are roughly 
classified into: a top copy-from volume VI that does not 
have a copy-from volume, and that has a copy-to volume; a 
middle copy-to volume V2 that has a copy-from volume and a 
copy-to volume; and an end copy-to volume V3 that has a 
copy-from volume, and that does not have a copy-to volume. 
The relationship among the target volumes to be remote- 
copied forms a tree shape with the top copy-from volume VI 
being a vertex. Data to be remote-copied is transferred in 
succession from the top copy-from volume VI to the end copy- 
to volume V3 through the middle copy-to volume V2 . 
[0029] 

Fig. 1 illustrates a configuration in which a volume 
in the site 1 is specified as the top copy-from volume VI. 
In this configuration, a remote copy is made through the 
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middle copy-to volume V2 in the site 2 to the end copy-to 
volume V3 in the site (K - 1) and the end copy-to volume V3 
in the site N, and also a remote copy is made through the 
middle copy-to volume V2 in the site K to the end copy-to 
volume V3 in the site (N -1). 
[0030] 

Fig. 2 is a diagram illustrating a configuration of 
the storage system 2. A storage controller 21 comprises 
components of: channel interfaces 3 each of which is 
connected to the host computer 1; disk interfaces 4 each of 
which is connected to a storage device 22; duplexed 
nonvolatile management information memories 6 for storing 
management information; and duplexed nonvolatile cache 
memories 5 for storing data. The above components are 
connected to one another through paths. Each of the channel 
interfaces 3 and each of the disk interfaces 4 are connected 
to the duplexed management information memories 6 and also 
to the duplexed cache memories 5. In addition, the storage 
controller 21. further comprises a maintenance terminal 23 
used to instruct the storage controller 21 in question, and 
to display an internal state of the storage controller 21 in 
question . 
[0031] 

The channel interface 3 comprises: write processing 
31 for handling a write request received from the host 
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computer 1, and a write request for a remote copy received 
from another storage system 2 that is a source of the remote 
copy; bit-map update processing 32 for updating a bit map 
corresponding to a range within which a write has been made; 
asynchronous transmission processing 33 for, if the own 
storage system 2 is a source of an asynchronous remote copy, 
transferring write data to said another storage system 2 as 
a copy destination in asynchroni zat ion with a write request; 
bit-map switching start, processing 34 for triggering, from 
the storage system 2 having the top copy-from volume VI, 
switching of a bit map 809 covering all volumes that are 
copy destinations of the volume in question; bit-map 
switching processing 35 for receiving a bit-map switching 
request from the bit-map switching start processing 34, and 
then for switching the bit map; and difference copy 
processing 36 for, when reconfiguring a remote-copy 
configuration among remaining sites after suffering from 
disaster in an arbitrary site, making volume contents in 
each of the remaining sites coincident with those in the 
other remaining sites. 
[0032] 

The disk interface 4 comprises: asynchronous 
normalization processing 41 for, if the own storage system 2 
is a destination of an asynchronous remote copy, normalizing 
remote-copy data that is temporarily saved as temporary data 
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at the time of writing from a copy source; and bit-map 
update processing 32 that is the same as the bit-map update 
processing 32 on the side of the channel interface 3. 
[0033] 

There are a plurality of pieces of processing for 
each kind of the above processing. As a unit of existence, 
the bit-map switching start processing 34, the bit-map 
switching processing 35, and the difference copy processing 
36 may exist on a volume basis; or the asynchronous 
transmission processing 33 and the asynchronous 
normalization processing 41 may also exist for each write- 
sequence management information 62 described below. 
[0034] 

The cache memories 5 store data that is stored in the 
storage device 22, and that is read/written from/to the host 
computer 1. Each of the cache memories 5 can be broadly 
classified into a normal area and a temporary area. The 
reason why each cache memory 5 has the temporary area is 
that if the own storage system 2 is a source of an 
asynchronous remote copy, when untransf erred remote-copy 
data is written to a copy destination again, it is necessary 
to save untransf erred data before the write, or that if the 
own storage system 2 is a destination of an asynchronous 
remote copy, it is necessary to save remote-copy data 
existing after the remote-copy data is received from a copy 
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source before the remote-copy data is normalized by the 

asynchronous normalization processing 41. 

[0035] 

The management information memories 6 store 
management information required for the operation of the 
storage system 2. The management information includes 
remote-copy management information 61. The remote-copy 
management information 61 includes write-sequence management 
information 62 and volume management information 63. 
[0036] 

The write-sequence management information 62 is used 
to normalize remote-copy data in the write order received 
from the host computer 1 in an asynchronous remote copy. 
There are a plurality of pieces of the write-sequence 
management information 62. Each piece of write-sequence 
management information 62 includes: a write-sequence number 
counter 71 for writing the write order received from the 
host computer 1; a write-sequence management information 
entry 721 of a write-sequence management information entry 
area 72, which is assigned on a write data basis, and in 
which a write sequence number is saved so that the write 
sequence number is associated with write data; a 
transmission target write-sequence management information 73 
for, in a copy source of an asynchronous remote copy, 
registering untransf erred remote-copy data in a copy 
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destination; and normalization target write-sequence 
management information 74 for, in a copy destination of an 
asynchronous remote copy, registering remote-copy data that 
has been received from a copy source but has not yet been 
normalized . 
[0037] 

Fig. 3 is a diagram illustrating a structure of the 
write-sequence management information 62 and the 
relationship between the write-sequence management 
information 62 and the cache memory 5. The write-sequence 
management information entry 721 is assigned to each piece 
of write data. A location of the write data in the cache 
memory 5 is stored in the write-sequence management 
information entry 721. In addition, the transmission target 
write-sequence management information 73 and the 
normalization target write-sequence management information 
74 may also be configured to have a queue structure for 
connecting the write-sequence management information entry 
721. 
[0038] 

In addition, if it is necessary to ensure the write 
order received from the host computer 1 among a plurality of 
volumes, a write sequence number is assigned to write data 
in each of the plurality of volumes by use of the write- 
sequence number counter 71 of the same write-sequence 
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management information 62, and then the write sequence 
numbers are registered in the same transmission target 
write-sequence management information 73. In a copy- 
destination, the write sequence numbers are registered in 
the same normalization target write-sequence management 
information 74, and normalizations are made according to the 
write sequence numbers. 
[0039] 

Returning to Fig. 2, the volume management 
information 63 is management information about a target 
volume to be remote-copied. There are a plurality of pieces 
of the volume management information 63. Each piece of 
volume management information 63 comprises: a write-sequence 
management information number 801 for specifying write- 
sequence management information 62 used by an own volume; 
copy-to information 802 of the volume in question; copy-from 
information 803 of the volume in question; a bit-map 
switching flag 804 used to exclude bit-map switching: the 
latest write sequence number 805 used when the volume in 
question is a copy-to volume; a bit-map switching trigger 
write sequence number 806 which is compared with the latest 
write sequence number so as to trigger bit-map switching; a 
write-sequence-number check request flag 807 which is used 
to request at the time of bit-map switching to check whether 
or not the latest write sequence number 805 reaches the bit- 



map switching trigger write sequence number 806 in the own 
storage system 2; a wr ite-sequence-number reach flag 808 for 
indicating that the latest write sequence number 805 has 
reached the bit-map switching trigger write sequence number 
806; a plurality of bit maps 809, each of which stores 
updated points of the volume in question; a plurality of 
bit-map validity flags 810, each of which indicates that a 
corresponding bit map is valid; a plurality of bit-map write 
start write sequence numbers 811 each indicating the write 
sequence number from which writing to a corresponding bit 
map is started; a plurality of bit-map updated data amount 
counters 812 each indicating the amount of updated data of a 
corresponding bit map; and a difference bit map 813 which is 
used when it becomes necessary to make contents of a volume 
coincide with those in the other sites as a result of 
suffering from disaster. 
[0040] 

The copy-to information 802 and the copy-from 
information 803 include: information that is required to 
communicate with the copy-from or copy-to storage system 2; 
and information for identifying a copy-from or copy-to 
volume . 
[0041] 

According to the embodiment of the present invention, 
for the sake of simplification, management information is 
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provided on a volume basis (the volume management 
information 63). However, this does not limit the scope of 
the present invention. The management information can be 
provided in an arbitrary unit that can be shared between the 
storage systems 2. Accordingly, it is possible to make a 
remote copy in the arbitrary unit that can be shared between 
the- storage systems 2. The write processing 31 which is 
executed in each storage system 2 will be described with 
reference to Fig. 4. 
[0042] 

Here, a write requester and classification of write 
data, which are targeted by the write processing 31, will be 
outlined so that the description can be easily understood. 
The write requester is the host computer 1, or the storage 
system 2 that is a source of a remote copy. 
[0043] 

Depending on a write target volume, write data from 
the host computer 1 can be classified into: 1) write data 
having no remote-copy destination; and write data having a 
remote-copy destination. The write data having a remote- 
copy destination can be further classified into: 2) write 
data having only a synchronous remote-copy destination; 3) 
write data having only an asynchronous remote-copy 
destination; and 4) write data having both a synchronous 
remote-copy destination and an asynchronous remote-copy 
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destination . 
[0044] 

Depending on a write target volume, write data from 
the storage system 2 which is a remote-copy source can be 
classified into: write data that is written by a synchronous 
remote copy; and write data that is written by an 
asynchronous remote copy. As is the case with the above 1) 
through 4), the write data that is written by a synchronous 
remote copy can be classified into: 5) write data having no 
remote-copy destination; and write data having a remote-copy 
destination. The write data having a remote-copy 
destination can be further classified into: 6) write data 
having only a synchronous remote-copy destination; 7) write 
data having only an asynchronous remote-copy destination; 
and 8) write data having both a synchronous remote-copy 
destination and an asynchronous remote-copy destination. In 
addition, as is the case with the above 1), 4), the write 
data which is written by an asynchronous remote copy can be 
classified into: 9) write data having no remote-copy 
destination; and 10) write data having only an asynchronous 
remote-copy destination. The reason why 2) and 3) are 
excluded from the targets is that in the synchronous remote 
copy, a write completion notification is transferred to the 
host computer on the completion of a remote copy. 
Accordingly, even if, in the middle copy-to volume V2 , 
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remote-copy data received by an asynchronous remote copy is 
remote-copied to a volume in another site by a synchronous 
remote copy, it is a meaningless and unrealistic matter. 
[0045] 

In the write processing 31, first of all, a write 
request is received (step S401), and then a judgment is made 
as to whether or not conditions that a write requester is 
the host computer 1, and that a write target volume is not a 
remote-copy source, hold true (step S402). 
[0046] 

If both of the conditions in the step S402 hold true, 
the above write request is a write request from the host 
computer 1 to a volume that does not relate to a remote copy 
(more specifically, write data is equivalent to the 1) 
described above) . Accordingly, write data is stored in a 
normal area of the cache memory 5 (step S403), and then the 
write requester is notified of write completion (step S404). 
The storage controller 21 writes this write data to the 
storage device 22 in asynchronization with the write request 
from the host computer 1. 
[0047] 

If one or both of the conditions in the step S402 
does not hold true, a check is made as to whether or not the 
write requester is the host computer 1 (step S405) . 
[0048] 
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If the condition in the step S405 holds true (more 
specifically, the write data is equivalent to any of 2), 3), 
4) described above), the wr ite-sequence management 
information entry 721 to be associated with the write data 
is created, and then a write sequence number is given to the 
write-sequence management information entry 721. To be more 
specific, with reference to the write-sequence management 
information number 801 of the volume management information 
63 corresponding to a volume to which a write has been made 
from the host computer 1, the write-sequence management 
information entry 721 of the write-sequence management 
information 62 corresponding to the write-sequence 
management information number 801 is created. After that, a 
value of the write-sequence number counter 71 is set, and 
then the value of the write-sequence number counter 71 is 
added so that the write-sequence management information 
entry 721 is associated with the write data. 
[0049] 

The addition of the write sequence number is executed 
regardless of synchronization/asynchronization of the remote 
copy. This is because even if direct copy destinations of 
the volume in question include only synchronous remote-copy 
destinations, a further destination of these copy 
destinations may execute an asynchronous remote copy. 
[0050] 
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If the condition in the step S405 does not hold true 
(more specifically, the write data is equivalent to any of 
6) through 10) described above), the write-sequence 
management information entry 721 to be associated with the 
write data is created, and then the write-sequence 
management information 62 including a write sequence number, 
which has been received together with the write data, is 
stored (step S407 ) . 
[0051] 

Next, the write data is stored in the cache memory 5. 
A method for storing the write data in the cache memory 5 
depends on a write target volume. There are three cases as 
follows: A) a case where the write target volume is an 
asynchronous remote-copy destination (more specifically, 
equivalent to any of 9), 10) described above); B) a case 
where the write target volume is not an asynchronous remote- 
copy destination but an asynchronous remote-copy source 
(more specifically, equivalent to any of 4), 7), 8) . 
described above) ; and C) a case other than the above A) , B) 
(more specifically, equivalent to any of 2), 5), 6) 
described above) . This is because in the case of A) it is 
necessary to temporarily store received write data, and then 
formally write the write data according to the write 
sequence number. This is also because in the case of B) if 
untransf erred write data is written to an asynchronous 



remote-copy destination, it is necessary to independently 
store this write data until old write data is transferred. 
[0052] 

First of all, a judgment is made as to whether or not 
a write target volume is an asynchronous remote-copy 
destination (step S408). 
[0053] 

If the condition in the step S4.08 holds true (more 
specifically, equivalent to A) described above) , it is 
necessary to write the write data in question according to a 
write sequence number indicating the update order from the 
host computer 1. Accordingly, the write data is temporarily 
stored in the temporary area of the cache memory 5 (step 
S409) . Then, the write-sequence management information 
entry 721 corresponding to the write data is registered in 
the normalization target write-sequence management 
information 74 as target data to be normalized (step S410) . 
In Fig. 3, write-sequence management information entries 
721_e, 721_f correspond to the target data to be normalized. 
[0054] 

Moreover, a judgment is made as to whether or not a 
write target volume is a copy source of an asynchronous 
remote copy (step S411) . If the condition in the step S411 
holds true (more specifically, equivalent to any of 3), 4), 
7), 8), 10) described above), the write-sequence management 
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information entry 721 corresponding to the write data is 
registered in the transmission target write-sequence 
management information 73 as target data to be transferred 
(step S412) . In Fig. 3, write-sequence management 
information entries 721_b, 721_c, 721_d, 721_e correspond to 
the target data to be transferred. 
[0055] 

If the condition in the step S408 holds true, a 
further judgment is made as to whether or not the write 
target volume is a copy source of an asynchronous remote 
copy (step S4 13 ) . 
[0056] 

If the condition in the step S413 holds true (more 
specifically, equivalent to B) described above) , write data 
that has not yet been transferred to an asynchronous remote- 
copy destination is stored in the temporary area of the 
cache memory 5, and then the write data is stored in the 
normal area of the cache memory 5 (step S414). In Fig. 3, 
the write-sequence management information entries 721_b, 
721_c, 721_d correspond to the write data. Further, the 
bit-map update processing 32 is executed (step S415) . The 
bit-map update processing 32 will be described later. After 
that, the process proceeds to the step 411. 
[0057] 

If the condition in the step S413 does not hold true 



(more specif ically, equivalent to C) described above) the 
write data in question is neither a copy destination of an 
asynchronous remote copy nor a copy source of the 
asynchronous remote copy. Accordingly, the write data is 
stored in the normal area of the cache memory 5 (step S416) . 
In Fig. 3, wr i te-sequence management information entries 
721_a correspond to the write data. Further, the bit-map 
update processing 32 is executed (step S417) . The bit-map 
update processing 32 will be described later. After that, 
the process proceeds to the step 418. 
[0058] 

Next, a judgment is made as to whether or not the 
write target volume is a copy source of a synchronous remote 
copy (step S418) . If the condition in the step S418 holds 
true (more specifically, equivalent to any of 2), 4), 6), 8) 
described above) , write data and contents of the write- 
sequence management information entry 721 are transferred to 
a copy destination of the synchronous remote copy (step 
S419) . With the object of shortening the time taken after 
receiving a write request before a write completion 
notification is transferred to the write requester, the 
synchronous remote copy in question may also be executed in 
parallel with respect to a plurality of copy destinations. 
[0059] 

Next, a judgment is made as to whether the write- 



sequence management information 721 corresponding to the 
write data in question is registered in neither the 
transmission target write-sequence management information 73 
nor the normalization target write-sequence management 
information 74 (step S420) . If the condition in the step 
S420 holds true (more specifically, equivalent to any of 2) , 
5), 6) described above) , the write-sequence management 
information entry 721 corresponding to the write data in 
question is not necessary. Therefore, the write-sequence 
management information entry 721 is released (step S421), 
which corresponds to a write-sequence management information 
entry 721_g in Fig. 3. 
[0060] 

Lastly, a write completion notification is 
transferred to the write requester (step S404), before the 
processing ends. 
[0061] 

The asynchronous transmission processing 33 which is 
executed in a storage system 2 having a copy-from volume of 
an asynchronous remote copy among the storage systems 2 will 
be described with reference to Fig. 5. This processing may 
also be executed for each write-sequence management 
information 62. 
[0062] 

First of all, a target to be transferred is selected 



from among the write-sequence management information entries 
721 registered in the transmission target write-sequence 
management information 73 (step S501) . 
[0063] 

Next, the selected write-sequence management 
information entry 721 and write data corresponding to the 
write-sequence management information entry 721 in question 
are transferred to an asynchronous remote-copy destination 

(step S502) . 

[0064] 

Next, the transferred write-sequence management 
information entry 721 is deleted from the transmission 
target write-sequence management information 73 (step S503) . 
[0065] 

Next, a judgment is made as to whether or not the 
transferred write-sequence management information entry 721 
is registered also in the normalization target write- 
sequence management information 74 (step S504). If the 
condition in the step S504 does not hold true, management as 
target data of the asynchronous remote copy becomes 
unnecessary. Accordingly, the transferred write-sequence 
management information entry 721 is released (step S505) . 
Here, if the transferred write data has been stored in the 
temporary area of the cache memory 5, the temporary area in 
question is also released (step S506) . 



[0066] 

The asynchronous normalization processing 41 which is 
executed in a storage system 2 having a copy-to volume of an 
asynchronous remote copy among the storage systems 2 will be 
described with reference to Fig. 6. This processing may 
also be executed for each wr ite-sequence management 
information 62. 
[0067] 

First of all, with reference to the write-sequence 
management information entry 721 registered in the 
normalization target write-sequence management information 
74, a range within which write sequence numbers successively 
continue is selected as a target to be normalized (step 
S601) . 
[0068] 

Next, a judgment is made as to whether or not all 
write data which is the selected target to be normalized has 
already been normalized (step S602). 
[0069] 

If the condition in the step S602 holds true, the 
process proceeds to a step S606. 
[0070] 

If the condition in the step S602 does not hold true, 
among the pieces of write data which have been selected as 
the target to be normalized, a piece of write data 
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corresponding to the oldest write sequence number is 
transferred from the temporary area to the normal area in 
the cache memory 5, and then the piece of write data is 
normalized there (step S603) . 
[0071] 

Next, the normalized write-sequence management 
information entry 721 is deleted from the normalization 
target write-sequence management information 74 (step S604). 
Next, the bit-map update processing 32 is executed (step 
S605) . The bit-map update processing 32 will be described 
later. After that, the process returns to the step S602. 
[0072] 

Next, a judgment is made as to whether or not the 
normalized write-sequence management information entry 721 
is registered also in the transmission target write-sequence 
management information 73 (step S606) . If the condition in 
the step S606 does not hold true, management as target data 
of the asynchronous remote copy becomes unnecessary. 
Accordingly, the normalized write-sequence management 
information entry 721 is released (step S607). In addition, 
the temporary area of the cache memory 5, in which the 
normalized write data has been stored, is released (step 
S608) . 
[0073] 

The bit-map switching start processing 34 which is 
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executed in a storage system 2 having the top copy-from 
volume VI among the storage systems 2 will be described with 
reference to Fig. 7. This processing may also be executed 
on a volume basis if a value of an updated data amount 
counter 812 of the bit map 809 whose bit-map validity flag 
810 is in an ON state exceeds a fixed threshold value (if 
the number of bit maps 809 whose bit-map validity flag 810 
is in an ON state is two or more, a bit map whose write 
start write sequence number 811 is the oldest is selected 
from among them) . 
[0074] 

In addition, if the elapsed time after the lastly 
executed bit map switching exceeds a fixed threshold value, 
the bit-map switching start processing 34 may also be 
executed on a volume basis. 
[0075] 

First of all, for the top copy-from volume VI, all 
middle copy-to volumes V2 corresponding to the top copy-from 
volume VI, and all end copy-to volumes V3 corresponding to 
the top copy-from volume VI, in order to prevent bit map 
switching from being newly started before the last bit map 
switching ends, a judgment is made as to whether or not the 
bit-map switching flag 804 is in an OFF state (step S701) . 
[0076] 

If the condition in the step S701 does not hold true, 



the bit map switching is still being executed. Therefore, 

the processing is ended. 

[0077] 

If the condition in the step S701 holds true, the 
bit-map switching flag 804 is brought into an ON state (step 
S702) . 
[0078] 

Next, a write sequence number used to trigger bit-map 
switching is determined. Then, the bit-map switching 
trigger write sequence number 806 of the management 
information memory is set at the determined write sequence 
number (step S703) . A method for determining a write 
sequence number used to trigger bit-map switching may be the 
following: A write sequence number is so generated that the 
latest write sequence number 805 relating to the middle 
copy-to volume V2 and the end copy-to volume V3 does not 
exceed the generated write sequence number before a 
notification is transferred from the top copy-from volume VI 
to all middle copy-to volumes V2 corresponding to the top 
copy-from volume VI, and further to all end copy-to volumes 
V3 corresponding to the top copy-from volume VI. This write 
sequence number is used as the bit-map switching trigger 
write sequence number 806. 
[0079] 

In addition, it is intended that the bit-map 



switching trigger write sequence number 806 becomes a write 
start write sequence number of a bit map 809 to which bit- 
map switching is made. However, if write sequences of a 
plurality of volumes are managed with a piece of write- 
sequence management information 62, the determined bit-map 
switching trigger write sequence number 806 is not always 
assigned to write data of the volume in question. However, 
a write sequence number that is first assigned to the volume 
in question with the determined bit-map switching trigger 
write sequence number 806 being exceeded is uniquely 
determined for the top copy-from volume VI, all middle copy- 
to volumes V2 corresponding to the top copy-from volume VI, 
and all end copy-to volumes V3 corresponding to the top 
copy-from volume VI. Therefore, the write sequence number 
that is first assigned to the volume in question with the 
bit-map switching trigger write sequence number 806 being 
exceeded can be used as a write start write sequence number 
of the bit map 809 to which bit-map switching is made. 
[0080] 

Next, the write-sequence-number reach flag 808 is 
brought into an OFF state (step S704). This write-sequence- 
number reach flag 808 is used to indicate that the latest 
write sequence number 805 becomes larger than or equal to 
the bit-map switching trigger write sequence number 806. 
[0081] 



Next, the write-sequence-number check request flag 
807 is brought into an ON state (step S705) . This write- 
sequence-number check request flag 807 is used to request 
the bit-map update processing 32 to compare the latest write 
sequence number 805 with the bit-map switching trigger write 
sequence number 806. 
[0082] . 

Next, the bit-map switching trigger write sequence 
number 806 is transferred to all direct copy destinations of 
the volume in question to request a wri te-sequence-number 
reach notification when the latest write sequence number 
exceeds the bit-map switching trigger write sequence number 
806 (step S706) . 
[0083] 

Next, judgments are made as to whether or not the 
write-sequence-number reach flag 808 of the volume in 
question is ON, and whether or not write-sequence-number 
reach notifications are received from all of the direct copy 
destinations of the volume in question (step S707) . If one 
or both of the conditions in the step S707 does not hold 
true, after waiting for a fixed period of time (step S708), 
the judgments in the step S707 are made again. It is to be 
noted that notifications from the direct copy destinations 
of the volume in question are successively stored in the 
copy-to information 802. 



[0084 ] 

If both of the conditions in the step S707 hold true, 
a validity flag 810 of a bit map 809 corresponding' to the 
oldest write start write sequence number 811 is brought into 
an OFF state, said bit map 809 being selected from among the 
bit maps 809 in the own site. Then, the bit map 809 in 
question is cleared (step S709) . This is because at this 
point of time, for the top copy-from volume VI, all middle 
copy-to volumes V2 corresponding to the top copy-from volume 
VI, and all end copy-to volumes V3 corresponding to the top 
copy-from volume VI, the latest write sequence numbers 805 
of these volumes exceed the bit-map switching trigger write 
sequence number 806, and consequently writing to a new bit 
map 809 has already been started. 
[0085] 

Next, all of the direct copy destinations of the 
volume in question are requested to perform bit map 
switching (step S710) . 
[0086] 

Next, a judgment is made as to whether or not bit map 
switching completion notifications are received from all of 
the direct copy destinations of the volume in question (step 
S711) . If the condition in the step S711 does not hold true, 
after waiting for a fixed period of time (step S712), the 
judgment in the step S711 is made again. 



[0087] 

If the condition in the step S711 holds true, the bit 
map switching is completed. Therefore, the bit-map 
switching flag 804 is brought into an OFF state (step S713), 
before the processing ends. 
[0088] 

The bit-map switching processing 35 which is executed 
in a storage system 2 having the middle copy-to volume V2 
and the end copy-to volume V3 among the storage systems 2 
will be described with reference to Fig. 8. 
[0089] 

First of all, the bit-map switching trigger write 
sequence number 806 is received from a copy-from site. Then, 
the bit-map switching trigger write sequence number 806 of 
the middle copy-to volume V2, and that of the end copy-to 
volume V3, are set at a value of the received bit-map 
switching trigger write sequence number 806 (step S801) . 
Here, the middle copy-to volume V2 and the end copy-to 
volume V3 are targets of bit map switching. 
[0090] 

Next, the write-sequence-number reach flag 808 is 
brought into an OFF state (step S802), and the write- 
sequence-number check request flag 807 is brought into an ON 
state (step S803) . 
[0091] 



Next, a judgment is made as to whether or not a 
direct copy destination exists in the volume in question 
(step S804 ) . 
[0092] 

If the condition in the step S804 does not hold true 
(more specifically, the volume in question is equivalent to 
the end copy-to volume V3 ) , the process proceeds to a step 
S808. 
[0093] 

If the condition in the step S804 holds true (more 
specifically, the volume in question is equivalent to the 
middle copy-to volume V2 ) , the bit-map switching trigger 
write sequence number 806 is transferred to all direct copy 
destinations of the volume in question to request a write- 
sequence-number reach notification when the latest write 
sequence number 805 exceeds the bit-map switching trigger 
write sequence number 806 (step S805). 
[0094] 

Next, a judgment is made as to whether or not write- 
sequence-number reach notifications are received from all of 
the direct copy destinations of the volume in question (step 
S806) . If the condition in the step S806 does not hold true, 
after waiting for a fixed period of time (step S807), the 
judgment in the step S806 is made again. 
[0095] 



If the condition in the step S806 holds true, a 
judgment is made as to whether or not the write-sequence- 
number reach flag of the volume in question is ON (step 
S808). If the condition in the step S808 does not hold true, 
after waiting for a fixed period of time (step S809) , the 
judgment in the step S808 is made again. 
[0096] 

If the condition in the step S808 holds true, the 
direct copy source of the volume in question is notified 
that the write sequence number has been reached (step S810). 
[0097] 

Next, a judgment is made as to whether or not a bit- 
map switching request has been received from a direct copy 
source of the volume in question (step S811). If the 
condition in the step S811 does not hold true, after waiting 
for a fixed period of time (step S812), the judgment in the 
step S811 is made again, 
[0098] 

If the condition in the step S811 holds true, the 
bit-map validity flag 810 of a bit map 809 corresponding to 
the oldest write start write sequence number 811 is brought 
into an OFF state, the bit map 809 being selected from among 
the bit maps 809 of the volume in question. Then, the bit 
map 809 in question is cleared (step S813). 
[0099] 



Next, a judgment is made as to whether or not a 
direct copy destination exists in the volume in question 
(step S814 ) . 
[0100] 

If the condition in the step S814 does not hold true 
(more specifically, the volume in question is equivalent to 
the end copy-to volume V3) , the process proceeds to a step 
S818. 
[0101] 

If the condition in the step S814 holds true (more 
specifically, the volume in question is equivalent to the 
middle copy-to volume V2 ) , all of the direct copy 
destinations of the volume in question are requested to 
perform bit-map switching (step S815). 
[0102] 

Next, a judgment is made as to whether or not bit map 
switching completion notifications are received from all of 
the direct copy destinations of the volume in question (step 
S816) . If the condition in the step S816 does not hold true, 
after waiting for a fixed period of time (step S817), the 
judgment in the step S816 is made again. 
[0103] 

If the condition in the step S816 holds true, the 
direct copy source of the volume in question is notified 
that the bit map switching has been completed (step S818), 



and then the processing ends. 
[0104] 

The bit-map update processing 32, which is executed 
in the write processing 31 and the asynchronous 
normalization processing 41, will be described with 
reference to Fig . 9 . 
[0105] 

First of all, each bit of a bit map 809 whose bit-map 
validity flag 810 is ON is brought into an ON state, the 
said bit corresponding to write data, and then the updated 
data amount counter 812 corresponding to the updated bit map 
809 is incremented (step S901). 
[0106] 

Next, the latest write sequence number 805 is updated 
to a write sequence number corresponding to the write data 
in question (step S902). 
[0107] 

Next, a judgment is made as to whether or not the 
write-sequence-number check request flag 807 is ON (step 
S903) . 
[0108] 

If the condition in the step S903 does not hold true, 
the processing ends . 
[0109] 

If the condition in the step S903 holds true, a 
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judgment is made as to whether or not the latest write 
sequence number 805 is larger than or equal to the bit-map 
switching trigger write sequence number 806 (step S904) . 
[0110] 

If the condition in the step S904 does not hold true, 
the processing ends. 
[0111] 

If the condition in the step S904 holds true, the 
write-sequence-number reach flag 808 is brought into an ON 
state (step S905) . 
[0112] 

Next, the bit-map validity flag 810 of a bit map 809 
whose bit-map validity flag 810 is OFF is brought into an ON 
state, and then the write start write sequence number 811 is 
set at a value of the latest write sequence number 805 (step 
S906) . 
[0113] 

Next, each bit of the bit map 809 whose bit-map 
validity flag 810 has been newly brought into the ON state 
is switched to ON, the said bit corresponding to write data, 
and then the updated data amount counter 812 corresponding 
to the updated bit map 809 is incremented (step S907) . 
[0114] 

Lastly, the write-sequence-number check request flag 

r- 

807 is brought into an OFF state (step S908), before ending 
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the processing. 
[0115] 

Fig. 10 is a diagram schematically illustrating 
processing in each site, and communications between sites, 
at the time of bit-map switching. 
[0116] 

The bit-map switching start processing 34 is executed 
in a site having the top copy-from volume VI; and the bit- 
map switching processing 35 is executed in a site having the 
middle copy-to volume V2 or the end copy-to volume V3 . 
However, the bit-map update processing 32 is partially 
included. An appropriate step number is given to each step 
to be executed. 
[0117] 

Fig. 10 schematically illustrates bit-map switching 
from a bit map 1 to a bit map 2 in the whole site. After 
that, the bit-map switching is alternately executed from the 
bit map 2 to the bit map 1, and further from the bit map 1 
to the bit map 2. 
[0118] 

By shifting the bit-map switching as the whole site 
from a state in which a single bit map is updated, through a 
state in which double bit maps are temporarily updated, to a 
state in which a single bit map is updated again, the write 
start write sequence number 811 can always share the same 



bit map 809 between arbitrary sites during an arbitrary 
moment including the bit-map switching, and thereby volume 
contents can be made to coincide with those in the site in 
which the write sequence number goes ahead. 
[0119] 

In addition, in the event that a failure is detected 
in a certain site during the bit-map switching, transfer of 
the bit-map switching trigger write sequence number 806 to 
the storage system 2 in each site may fail. In this case, 
in the storage system 2 having the top copy-from volume VI, 
the bit map 809 is not cleared in the step S709. Then, in a 
step S710, by requesting storage systems 2 in the other 
sites to transfer a bit-map switching completion 
notification without clearing the bit map 809, it is 
possible to continue management of differences including the 
site where the failure has occurred. 
[0120] 

Moreover, if each storage system 2 has triple bit 
maps 809 and a failure is detected in a certain site, as 
described above, by preventing the bit map 809 used when the 
failure has been detected from being cleared, and by 
continuing the bit-map switching by use of remaining bit 
maps, it is possible to continue the management of the 
differences including the site where the failure has 
occurred, and thereby to prevent the differences between 
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normal sites from increasing. Even if the number of sites 
in which a failure has occurred increases, by keeping 
unchanged the bit map 809, which is prevented from being 
cleared, until all sites become normal, it is possible to 
continue the management of the differences including the 
site where the failure has occurred. 
[0121] 

In addition, by transferring and receiving usual 
remote-copy data together with the contents of 
communications between two storage systems performed at the 
time of the bit-map switching, it is possible to reduce a 
communication load between the sites. 
[0122] 

Steps of making the volume contents coincide between 
sites after one of the sites suffers from disaster will be 
described with reference to Fig. 11. 
[0123] 

First of all, as a new direct copy source of an own 
volume that has lost a direct copy source as a result of the 
disaster from which the site has suffered, a volume whose 
latest write sequence number 805 is larger than that of the 
own volume is selected from among volumes having the same 
top copy-from volume VI as that included in the own volume 
(step S1101) . 
[0124 ] 
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Next, a transfer path is provided between the storage 
system 2 in an own site and the storage system 2 in a new 
direct copy-from site (step S1102) . 
[0125] 

Next, remote copying is started between the storage 
system 2 in the own site and the storage system 2 in the new 
direct copy-from site (step S1103) . Thereafter, updates 
made from the new direct copy-from site to the copy-from 
volume are successively reflected in a copy-to volume in the 
own site. 
[0126] 

Lastly, the difference copy processing 36 between the 
storage system 2 in the own site and* the storage system 2 in 
the new direct copy-from site is started (step S1104) . The 
difference copy processing 36 will be described later. 
[0127] 

By continuing remote copying also after completion of 
the difference copy processing 36, the remote copying 
becomes usual remote copy operation. 
[0128] 

As the new direct copy-from site described above, a 
site on the upstream side whose distance to the top copy- 
from volume VI is closer than the distance from the own site 
to the top copy-from volume VI may also be registered 
beforehand for the own volume. 



[0129] 

The difference copy processing 36 which is one of the 
steps of making the volume contents coincide between sites 
after one of the sites suffers from disaster will be 
described with reference to Fig. 12. 
[0130] 

First of all, a bit map 809 whose bit-map validity 
flag 810 is ON in both an own site and a new direct copy- 
from site is selected in the new direct copy-from site. 
Here, if there are a plurality of bit maps each satisfying 
this condition, a bit map 809 whose write start write 
sequence number 811 is the newest is selected from among the 
bit maps. Then, contents of the selected bit map 809 is 
copied to the difference bit map 813 that is used to 
determine data to be copied in the difference copy 
processing 36 (step S1201) . The reason why the bit map 809 
is copied to the difference bit map 813 is the following: 
In the new direct copy-from site, the remote copying may be 
continued even after the site has suffered from disaster, 
and as a result, the bit map 809 whose bit-map validity flag 
810 is ON may be continuously updated. This makes it 
necessary to save the differences between current volumes in 
both sites so as to prevent the amount of differences to be 
copied from increasing. 
[0131] 



Next, part corresponding to bits whose value is ON in 
the difference bit map 813 is copied from a copy-from volume 
in the new direct copy-from site to a copy-to volume in the 
own site (step S1202) . 
[0132] 

Lastly, the difference bit map 813 is cleared (step 
S1203), and then the processing ends. 
[0133] 

[Effects of the Invention] 

According to the present invention, in remote copies 
made among storage controllers that are located in N sites 

(the number of sites is in general three or more) , after one 
site suffers from disaster, it is possible to manage 
differences that are used to quickly make remote-copy data 
coincide with each other among the remaining sites. In 
addition, it is possible to keep the amount of information 
used to manage the differences constant without depending on 
the number of sites N among which remote copies are made. 

[Brief Description of the Drawings] 

[Fig. 1] 

Fig. 1 is a diagram illustrating how remote copies 
among N sites are made according to an embodiment of the 
present invention by way of example. 
[Fig. 2] 

Fig. 2 is a diagram illustrating a configuration of a 
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storage system according to an embodiment of the present 
invention by way of example. 
[Fig. 3] 

Fig. 3 is a diagram illustrating the relationship 
between wr ite-sequence management information and a cache 
memory according to an embodiment of the present invention 
by way of example. 
[Fig. 4] 

Fig. 4 is a flowchart illustrating an example of 
write processing that is executed in a storage system 
according to an embodiment of the present invention. 
[Fig. 5] 

Fig. 5 is a flowchart illustrating an example of 
asynchronous transmission processing that is executed in a 
storage system of an asynchronous remote copy source 
according to an embodiment of the present invention. 
[Fig. 6] 

Fig. 6 is a flowchart illustrating an example of 
asynchronous normalization processing that is executed in a 
storage system of an asynchronous remote copy destination 
according to an embodiment of the present invention. 
[Fig. 7] 

Fig. 7 is a flowchart illustrating an example of bit- 
map switching start processing that is executed in a storage 
system having a top copy-from volume VI according to an 
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embodiment of the present invention. 
[Fig. 8] 

Fig. 8 is a flowchart illustrating an example of bit- 
map switching processing that is executed in a storage 
system having a middle copy-from volume/an end copy-from 
volume according to an embodiment of the present invention. 
[Fig. 9] 

Fig. 9 is a flowchart illustrating an example of bit- 
map update processing which is executed in write processing 
and asynchronous normalization processing according to an 
embodiment of the present invention. 
[Fig. 10] 

Fig. 10 is a diagram schematically illustrating bit- 
map switching according to an embodiment of the present 
invention . 
[Fig. 11] 

Fig. 11 is a flowchart illustrating steps of making 
the volume contents coincide between sites after one of the 
sites suffers from disaster according to an embodiment of 
the present invention by way of example. 
[Fig. 12] 

Fig. 12 is a flowchart illustrating an example of 
difference copy processing which is one of the steps of 
making the volume contents coincide between sites after one 
of the sites suffers from disaster according to an 
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embodiment of the present invention. 
[Explanations of Symbols and Reference Numerals] 

1 ... Host computer, 2 ... Storage system, 21 ... 
Storage controller, 22 ... Storage device, 23 ... 
Maintenance terminal, 3 ... Channel interface, 31 ... Write 
processing, 32 ... Bit-map update processing, 33 ... 
Asynchronous transmission processing, 34 ... Bit-map 
switching start processing, 35 ... Bit-map switching 
processing, 36 ... Difference copy processing, 4 ... Disk 
interface, 41 ... Asynchronous normalization processing, 
5 ... Cache memory, 6 ... Management information memory, 
62 ... Write-sequence management information, 63 ... Volume 
management information, 71 ... Write-sequence number counter, 
721 . . . Write-sequence management information entry, 73 ... 
Transmission target write-sequence management information, 
74 ... Normalization target write-sequence management 
information, 804 . . . Bit-map switching flag, 805 . . . Latest 
write sequence number, 806 . . . Bit-map switching trigger 
write sequence number, 807 . . . Wr ite-sequence-number check 
request flag, 808 . . . Write-sequence-number reach flag, 
809 . . . Bit map, 810 . . . Bit-map validity flag, 811 . . . Bit- 
map write start write sequence number, 812 . . . Bit-map 
updated data amount counter, 813 . . . Difference bit map, 
Tl ... Global network, T2 ... Local network, VI ... Top 
copy-from volume, V2 ... Middle copy-to volume, V3 ... End 



copy-to volume. 



Fig. 1 

^Remote-copy data 
Global network Tl 

Site 1 

Host computer 1_1 

Local network T2__2 

Storage system 2_1 

Top copy-from volume Vl_l 

Site 2 

Host computer 1 
Local network T2_2 
Storage system 2_2 
Middle copy-to volume V2_2 

Site (K-l) 

Host computer 1_(K-1) 
Local network T2_(K-1) 
Storage system 2_(K-1) 
End copy-to volume V3_(K-1) 

Site (N-l) 

Host computer 1__(N-1) 
Local network T2 (N-l) 
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Storage system 2_(N-1) 

End copy-to volume V3_(N-1) 

Site (N-2) 

Host computer l_(N-2) 
Local network T2_(N-2) 
Storage system 2_(N-2) 
End copy-to volume V3_(N-2) 

Site K 

Host computer 1_K 
Local network T2_K 
Storage system 2__K 
Middle copy-to volume V2_K 

Fig. 2 

2: Storage system 

21: Storage controller 

3: Channel interface 
31: Write processing 

31: Write processing 
32: Bit-map update processing 

32: Bit-map update processing 
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33: Asynchronous transmission processing 

33: Asynchronous transmission processing 
34: Bit-map switching start processing 

34: Bit-map switching start processing 
35: Bit-map switching processing 

35: Bit-map switching processing 
36: Difference copy processing 

36: Difference copy processing 

4: Disk interface 

41: Asynchronous normalization processing 
41: Asynchronous normalization processing 

32: Bit-map update processing 
32: Bit-map update processing 

22: Storage device 

23: Maintenance terminal 

5: Cache memory 
Normal area 
Temporary area 

6: Management information memory 

61: Remote copy management information 

62 1: Write-sequence management information 1 
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71: Write-sequence number counter 

72: Write-sequence management information entry area 
73: Transmission target write-sequence management 
information 

74: Normalization target write-sequence management 
information 

62 (J-l) : Write-sequence management information (J - 

63_1: Volume_l management information 

801: Write-sequence management information number 

802: Copy-to information 

803: Copy-from information 

804: Bit-map switching flag 

805: Latest write sequence number 

806: Bit-map switching trigger write sequence number 

807: Write-sequence-number check request flag 

808: Write-sequence-number reach flag 



809_1; 


Bit 


map_l 




810_1: 


Bit 


-map__l 


validity flag 


811_1: 


Bit 


-map__l 


write start write sequence number 


812_1: 


Bit 


-map_l 


updated data amount counter 


809_2: 


Bit 


map_2 




810_2 : 


Bit 


-map 2 


validity flag 


811_2 : 


Bit 


-map 2 


write start write sequence number 



812_2: Bit-map_2 updated data amount counter 
813: Difference bit map 

63 (M-l) Volume number (M-l) management information 
Fig. 3 

2: Storage system 

5: Cache memory 
Normal area 

Write data DO Write data Dl Write data D2_new 
Temporary area 

Write data D2_old Write data D4 Write data D5 

6: Management information memory 

61: Remote copy management information 

62_1: Write-sequence management information 1 

71: Write-sequence number counter 

72: Write-sequence management information entry area 
721__a: Write-sequence management information entry (write 
sequence number) 

721_b Write-sequence management information entry (write- 
sequence number) 

721_c Write-sequence management information entry (write- 
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sequence number) 

721__d Write-sequence management information entry (write- 
sequence number) 

721_e Write-sequence management information entry (write- 
sequence number) 

721 f Write-sequence management information entry (write- 
sequence number) 

721_g Write-sequence management information entry (no entry) 
73: Transmission target write-sequence management 
information 

74: Normalization target write-sequence management 
information 



62 (J-l) : Write-sequence management information (J-l) 



Fig. 4 
Start 

S401: Receive a write request. 

S402: Is it judged that a write requester is a host computer 
and a write target volume is not a remote-copy source? 
S403: Store write data in a normal area of a cache memory. 
S405: Is the write requester the host computer? 
S406: Create a write-sequence management information entry 
to be associated with the write data, and give a write 
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sequence number to the write-sequence management information 
entry. 

S407: Create a write-sequence management information entry 
to be associated with the write data, and store received 
write-sequence management information . 

S408: Is the write target volume an asynchronous remote-copy 
destination? 

S413: Is the write target volume an asynchronous remote-copy 
source? 

S416: Store the write data in the normal area of the cache 
memory . 

S417: Perform bit-map update processing. 

S414: Store write data, which has not yet been transferred 
to an asynchronous remote-copy destination, in a temporary 
area of the cache memory, and then store the write data in 
the normal area of the cache memory. 
S415: Perform bit-map update processing. 

S409: Store the write data in the temporary area of the 
cache memory. 

S410: Register the write-sequence management information 
entry as a target to be normalized. 

S411: Is the write target volume an asynchronous remote-copy 
source? 

S412: Register the write-sequence management information 
entry as a target to be transferred. 



S418: Is the write target volume a synchronous remote-copy 
source? 

S419: Transfer the write data and the write-sequence 
management information to a copy destination. 
S420: Is it judged that the write-sequence management 
information entry is neither a target to be transferred nor 
a target to be normalized? 

S421: Release the write-sequence management information 
entry . 

S404: Notify the write requester of the write completion. 
End 

Fig . 5 
Start 

S501: Select a target to be transferred from among write- 
sequence management information entries registered as 
targets to be transferred. 

S502: Transfer write data corresponding to a write-sequence 
management information entry to an asynchronous remote-copy 
destination . 

S503: Delete the transferred write-sequence management 
information entry that has been registered as the target to 
be transferred. 

S504: Is the transferred write-sequence management 



I 



67 



information entry a target to be normalized? 

S505: Release the transferred write-sequence management 

information entry. 

S506: If a temporary area of a cache memory has been used by 
the transferred write data, release the temporary area. 
End 

Fig. 6 
Start 

S601: With reference to a write-sequence management 
information entry registered as a target to be normalized, 
select, as a target to be normalized, write data 
corresponding to write-sequence management information 
entries in a range within which write sequence numbers 
successively continue. 

S602: Have all pieces of write data that have been selected 
as a target to be normalized been normalized? 

S603: Among the pieces of write data that have been selected 
as the target to be normalized, transfer a piece of write 
data corresponding to the oldest write sequence number from 
the temporary area to the normal area in the cache memory. 
S604: Delete the normalized write-sequence management 
information entry that has been registered as the target to 
be normalized. 



I 



68 



S605: Perform bit-map update processing. 

S606: Is the normalized write-sequence management 

information entry a target to be transferred? 

S607: Release the normalized write-sequence management 

information entry. 

S608: Release a cache memory's temporary area that has been 

used by the normalized write data. 

End 

Fig. 7 
Start 

S701: Is a bit-map switching flag in an OFF state? 
S702: Bring the bit-map switching flag into an ON state. 
S703: Determine a write sequence number used to trigger bit 
map switching, and store the write sequence number in a 
management information memory. 

S704: Bring a write-sequence-number reach flag into an OFF 
state . 

S705: Bring a write-sequence-number check request flag into 
an ON state. 

S706: Transfer a bit-map switching trigger write sequence 
number to a direct copy destination of an own volume. 
S707: Is it judged that a write-sequence-number reach flag 
of the own volume is ON, and that write-sequence-number 
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reach notifications have been received from all of direct 
copy destinations of the own volume? 
S708: Wait for a fixed period of time. 

S709: Bring into an OFF state a validity flag of a bit map 
corresponding to the oldest write start write sequence 
number, and then clear the bit map, the bit map being 
selected from among bit maps of the own volume. 
S710: Request the direct copy destinations of the own volume 
to perform bit-map switching. 

S711: Have bit-map switching completion notifications been 
received from all of the direct copy destinations of the own 
volume? 

S712: Wait for a fixed period of time. 

S713: Bring the bit-map switching flag into an OFF state. 
End 



Fig. 8 
Start 

S801: Receive a bit-map switching trigger write sequence 
number from a direct copy source of an own volume, and store 
the bit-map switching trigger write sequence number in a 
management information memory. 

S802: Bring a wri te-sequence-number reach flag into an OFF 
state . 
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S803: Bring a write-sequence-number check request flag into 
an ON state. 

S804: Does a direct copy destination exist in the own 
volume? 

S805: Transfer a bit-map switching trigger write sequence 
number 806 to a direct copy destination of the own volume. 
S806: Have write-sequence-number reach notifications been 
received from all of direct copy destinations of the o : wn 
volume? 

S807: Wait for a fixed period of time. 

S808: Is a write-sequence-number reach flag of the own 
volume in an ON state? 

S809: Wait for a fixed period of time. 

S810: Notify the direct copy source of the own volume that 
the write sequence number has been reached. 

S811: Has a bit-map switching request been received from the 
direct copy source of the own volume? 
S812: Wait for a fixed period of time. 

S813: Bring into an OFF state a validity flag of a bit map 

corresponding to the oldest write start write sequence 

number, and then clear the bit map, the bit map being 

selected from among bit maps of the own volume. 

S814: Does a direct copy destination exist in the own 

volume? 

S815: Request the direct copy destinations of the own volume 
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to perform bit-map switching. 

S816: Have bit-map switching completion notifications been 
received from all of the direct copy destinations of the own 
volume? 

S817: Wait for a fixed period of time. 

S818: Notify the direct copy source of the own volume that 

the bit-map switching has been completed. 

End 

Fig . 9 

Start 

S901: Bring into an ON state each bit of a bit map whose 
validity flag is ON, the bit corresponding to write data, 
and then increment a value of an updated data amount counter. 
S902: Set the latest write sequence number at a write 
se q Uence number corresponding to the write data in question. 
S903: Is a write-sequence-number check request flag in an ON 
state? 

S904: Is the latest write sequence number larger than or 
equal to the bit-map switching trigger write sequence 
number? 

S905: Bring the write-sequence-number reach flag into an ON 
state . 

S906: Bring into an ON state a validity flag of a bit map 
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whose validity flag is OFF, and then set a write start write 
sequence number at a value of the latest write sequence 
number . 

S907: Bring into an ON state each bit of a bit map whose 
validity flag has been newly brought into an ON state, the 
bit corresponding to write data, and then increment a value 
of the updated data amount counter. 

S908: Bring the write-sequence-number check request flag in 

an OFF state. 

End 

Fig. 10 

VI: Top copy-from volume 
V2: Middle copy-to volume 
V3: End copy-to volume 

Bit map 1 Bit map 2 

Detect the timing in which the bit-map switching is 
triggered . 

S702: Bring the bit-map switching flag into an ON state. 

A state in which a validity flag is ON 
A state in which a validity flag is OFF 



S706: Transfer a bit-map switching trigger write sequence 
number . 

S904, S906, S907: When a bit map is updated, the latest 
write sequence number M is reached (> N) , and start an 
update to a bit map whose validity flag is OFF with a write 
start write sequence number. 

S805: Transfer a bit-map switching trigger write sequence 
number N. 

S904, S906, S907 When a bit map is updated, the latest write 
sequence number M is reached (> N) , and start an update to a 
bit map whose validity flag is OFF with a write start write 
sequence number. 

S904, S906, S907: When a bit map is updated, the latest 
write sequence number M is reached (> N) , and start an 
update to a bit map whose validity flag is OFF with a write 
start write sequence number. 

S808: Check whether or not the write sequence number N has 
been reached. 

S810: Notify that the write sequence number N has been 
reached . 

S808: Check whether or not the write sequence number N has 
been reached. 

S810: Notify that the write sequence number N has been 
reached . 

S707: Check whether or not the write sequence number N has 



been reached. 

S709: Bring into an OFF state a validity flag of a bit map 
corresponding to the oldest write start write sequence 
number, and then clear the contents of the bit map. 
S710: Request to perform bit-map switching. 

S813: Bring into an OFF state a validity flag of a bit map 
corresponding to the oldest write start write sequence 
number, and then clear the contents of the bit map. 
S815: Request to perform bit-map switching. 

S813: Bring into an OFF state a validity flag of a bit map 
corresponding to the oldest write start write sequence 
number, and then clear the contents of the bit map. 
S818: Notify that the bit-map switching has been completed. 
S818: Notify that the bit-map switching has been completed. 
S713: Bring the bit-map switching flag into an OFF state. 



Fig. 11 
Start 

S1101: As a new direct copy source of an own volume, select 
a volume whose latest write sequence number is larger than 
that of the own volume from among volumes having the same 
top copy-from volume as that included in the own volume. 
S1102: Provide a transmission path between a storage system 
in an own site and a storage system in a new direct copy- 
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from site. 

S1103: Start remote copying between the storage system in 
the own site and the storage system in the new direct copy- 
from site. 

S1104: Start difference copy processing between the storage 
system in the own site and the storage system in the new 
direct copy-from site. 
End 

Fig. 12 
Start 

S1201: Select, in a new direct copy-from site, a bit map 
whose validity flag is ON in both an own site and the new 
direct copy-from site, and if there are a plurality of bit 
maps each satisfying this condition, copy a bit map whose 
write start write sequence number is the newest to create a 
difference bit map. 

S1202: Copy an area corresponding to bits whose value is ON 
in the difference bit map from a copy source to a copy 
destination . 

S1203: Clear the difference bit map. 
End 
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-h^t-^it^it^if u\ ^^y^~ h 

alf-(C^Jt63l»»3a«*t4ESB«»J»KKa!l«-«PO 

^i4-^-r hA#, -y->r hB-cw^^ffJUtfttti^-f ^B 
dsi#or^tc^^ 0 *>f hB»^^t-J4, * 

hB-CK^H^1f3Hf^5*t>nSfc*!>, ^ hA-C 

[0 0 13] ^it. *M hW^tea-f-Sy^-half- 
^-^(D^tt^^A— B, A-^CO*J^, ^-f hA365««r* 

hB<bf->< hccDTff y ..-^'Ci^^^t- L 
y^—hatf-SrSM-f-sr t^M^ u\ 
im hA-B, A-cw<os»« c afflfW4«^ufc-y--r hA 

[0014] r<oraHic*fi6-r6fe«>. mmmzy*- 

itr*j<*«"Cfc5. b^t, ro*fe<o#^ -y--r 
h ^tcit^j L,Thhfr ££><gm l ^ rt*utf ft h ft v 

J: ? ft»je^o*l»r» Li^flt* t ft 

[0 0 15] 

o y *— h a if-^n^eafr&te, N-y--r hra-eos 
^ < i : s^cov^r%Ji:urv^ftl^ e 
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[0 0 16] ~(Dt^tf). NIM hmX*<DV*r— b^tr— 
M*«$U 1fc»L"O^*^Sfc0 0> (N- 

- h ^ e - £ WSfST # ft v * 5 R8II a* *> o fco 
[0 0 l 7] &5fc<o, 21M MB-Coy h = 

[0 0 18] #36930 Btttt* £{C3iJ-*f hJ£A_L<7)NlJ- 
[0 0 19] 

' — 9c v ^ — h ^ v-^&mzft^mmm^^v'&m 

tr-^y^-h^^ajeiSH^^s:. M*rBKi-« 
or. y h =3 tr-»jt«B, ztz&s a^-ifey* 

[0 0 2 0] £7c, y*-h3f— »^«5fflo3E»f«E 
BC»35»f $ixeiE«Kff ^^t 1 y h =i tr-*ft 



u y*-h = is.tu 5 , ^b°-5fey^- 

[0 0 2 1 ] «ffc/«cK*f»IS^^«]»)#x.*«t 

[0022] &&mm^xTj*mx~<nffi8i<D j $>') 

*fcW\ ^t°-5fey^~h=ib 0 -m^«5ffltc:M^1- 

[0023] -ztz. tzmmw^xT^mxtDmn^Q 
[0024] h<^8M ft^urv^ 

[0 0 2 5J tt, «***J:»), y^e-h^t: 0 -m^ 

$m. *3tfi, =t°-5fey t o -m^®iacoy 

y h = b°-5fe^s^^ftno y * 
- h = t o -5fc— <£> y h 3 t°-<nmn^ : g&mfe 

£Ii/^f A^^y^ h 3 3fc^<7> y -=e— h3t° 

[0 0 2 6] 

[0027] m i ic, N-y--< hffl-cooy h = tr-« 
^co^i- e #-y-^ Hcii, h^ytv-^ i 

J ^^OtEti^v'^xA2^$rLAN (Local 
Area Network) ^SAN (Storage 
Area Network) t V N o fc^ — 5/ h 

t7-^T2(c<to-cs^{c^urv^, -^r, #*y--r 
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10 0 2 81 V^r— b y a — A(D«^{i, 

-ft^ y ^-A^otf^ «^ p ^-^ v 2 , = 
v i &m&t-rz>y y-#<tft9, *£^t o -x;#y 

[0 0 2 9] mi -CIS, b 1 <Dt$ ]) ~ — t° 
-TC^^-^VUU t>f h2^f B ^t:'-^y 
a-i^V2^lT % b (K- 1) , W bNCD^ 

-^^-^V2^IT, b (N-l) <Dpfc#& = 
[0 0 3 0] gi2t^ I2it^®i/^XA2C0^^;^^ 

-to ^mm^m2 in, **h^>-¥~—? i 
2m<ti<ttMm(Dwmmm**v e t. 

'tZ2m<tl,tz7f»m(D*Wi<'~*^V St*/<*X* 
y 5^fteh,<t^LT^S 0 ffittftj^§£®2 l 

i*. ^i^i5it^w^@2 1 ^(D^^&izmmmw 
2 i^<D^m^m:^-r^tz^(D^{^M2 3*mA. 

[00 3 1] ^-t^/W 7^ — * 3 ii. b 

m<ntzfemw>'*'r&2frb<D}) b t°— cov^r b 
2, steti^g^^xA 2^^^ y h ^t^3 

»IE»I3 3, ^^fc°-7c4Cy ^--AV 1 ^^po 

s^sKy a-Ao^y b-^^s o gsr^t^s^tti 

t** b^:/^9#;tigib^Jl3 4. ey b-^ 



•>7fA2 y b ^ stx-fes^ 

Itio^f; y b fcT— ■ T 5 -* «riE^bi-**l^»!E 

h^:/M£r*Q<3I3 2$r{iI^^)o 

[0 0 3 3] ^^sii-tn-e^^^EL, *R£i-£¥ 

{ilS, try b^:y^9#;tigIW3l3 4, t^b^y 
^•9#^L^3 5, *5cfct>\ ^^^t°-^3 6^^ 

[0 0 3 4] ^3/^>*y5H IBttKB2 2JC1§ 

SBWi^f a 2 ^iuMfl y h atr-©3 
tf— 7cT-*>5^. =» t°-5fei^^5Hco y ^e— h =i t°- 
*f—9*mv>U YZMzM\^y4 hm<o*&tik J 7—* 

iHj^y^e- h =i (7>=i t 1 — 5fe-e*>6»^ ^t°-7cJ: 
*)g«U^IWI»3jE«;{kftl3l4 i^±o-cjE*<fc-r«*-c 

[0 0 3 5] tffflfMR^y 6tl. IS««Kv'^^A2 
6 2, ^y a -Aflffii6 3^^o 

[oo3 6] h->- tr^^mmme 2 

ffi-TSo 7>T bv'— ^^^^sit^e 2 u 
by 721, y h =3 e— © 

3 t°-5ctc:^v 3 t°-5feic^$5^<D y b 3 ^ — 7 s 
3, ^f^^y^— b=* Ma jfe{^*5V^T= tc 

[0 0 3 7] 0 3 1^ 7>f h^^fl*«6 2© 

Mt^t^^a^^ys^^to b^-^ 

^iC^^ hi/- 5r^^*a*«3iV b y 7 2 1 £rfJ9^ 



-5- 



Xs U Vhy 7 2 He*** 

5(DyJ hr- ^<ott@^rES:^i-So 

[0 0 3 8] *X h^yt^^ lj^feO)^ h 

y a-A|B"C«liELfel^»^lCWc, PC 
y4 Yis—trls^mHi&S 2<Dy-i hi/—^>-*#* 
7 1 £<£/BLTlg*tf>#y ~—J*<Dy-{ h^—? 

(C7>r vis—>r>*#$:m«) %x, mc&mtt&^'f h 

[0 0 3 9] H2(:IoT, 4fy zx. — -6,^31^6 3 

y Afiitf 6 3(i, s^y a -^Mt5 7^ 

2. ^B£#y »-A(0=fcT-5c««8 0 3, fy^y 

7^8 04, Mi^#y a -A^3 t°-$fc#y a-A-efc 

5^C0ft^7-r h^-7-^# 8 0 5, s^r^-r 

St'^ h-7y7 P D^#^i7>f 8 0 

6. fcTy h^^^«9#X.B#^g|5tS^@^xA 2iC 
*5l^T**f7>T h*>— 4r^# 8 0 5 £S tfy v^y^m 

o»SM*tS7^ h^y7#ii^7 7 ^8 

0 7, M7-f h^y^#8 0 5^t*y h-ry^ 
VmtLW&yJ hv-^y^#8 0 6tCi^L^cr <b £^ 

-f7>r tr^*#mmy?#s o 8, y$»#y^- 
^^77^8 1 o mm . mfc-rzKy v-w? 

y^Umm^y^ h^^#8 l l («*) > ttfo 

y h^y^S 1 3 *3&» 
[0 0 4 0] 3^Mi8 0 2, ^fcT— 5£tiM»8 0 3 

[00 4 1] fffi^cDfc^, *»W03M6<0^<e^ 



T\ f^1t^MiX^^A2rp^^- ; tT^6^C0WiT-y 

[0 0 4 2] RWSr»^l5«<i-6^A, 7^f hM3 
1 hK#5c£7>f hx-^^SSU^ 

1. £fcfi, y-e-h=it: 0 -7cco|21t^®i/^^A2r* 

[0 04 3] h=* yta- * l frb<D7^ hr- * 
tt, 7^^#y^l:J:0, l) y*— h=n*— 

5t<7)^v>t>^, y*— hair— 5fcwfiF5 2) mm 
y^bat°-5few<7)t)^ 3) #i^j$y ^— h ^ t° 

[0 0 4 4] y ^E— h 3 TeOISI&^v'.Xt^ 2fr 
b<D7^by*—*\*, 7>f h^y^-AICj;*), R 

jHy*-hatr-teJ:9 9>r h£ix*:kc\ ^^y^e 

h=» tr-fc:«fc0 h tstiizhcote, ±121) ~ 
4) £IrHHic, 5) ^-hn^-^ivNt^ y^e 
— h = &i><ox 6) |5]#j y ^e— h ^ tr— jfco 

t^tcia^cofi, ±|BD , 4) tlnl^, 9) y^e 

-h^^ot^'bo, i o) v ^r— h ^ 

ft^OfcOlc^t^a-So 2) , 3) ^MMtU 

^^^t°-3fe^y a-AV2ic*sv^ ^BOBy^-h 

[0045] h«3 i -m, 7-r h^^r 
yb°a-^ i , d>o, 7^h^^-^y*-h 

[0 0 4 6] ^T^S 4 0 2C0^^55:CO^ > 

^-^^co^^f hS*-C&S^i6 (±151) IdfcMf) • 

Uf5'7 6 S4 0 3) , h^T^ 
Ufy7'S4 04) o :^)7^h7-^it * 

mfPtt* 2 1 tftm&m 2 2 atf. 

[0 0 4 7]^T77 1 S4 0 2^ITO^ 7 
{**ry~7S 4 0 5) 0 

[0 0 4 8] ^7y7 a S4 0 5 <D0ki*f&&<D^ (±15 
2) 3) 4) ICfB^) . 7^hr-^l:^^*57^ 
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h^-^^fl^xy h y 7 2 1 £fl*»U h 

h->—y^»3ffi««^v h y 7 2 i £?fe(£u h 

>7#^>)y^7 KDffiSrflDffU 7^ hi/— ^^^^ 

h y 7 2 i &?>r h?*— *£BM*mao 

[0 0 4 9] 7^ h>'-^#«OWDfl D^~h=a 

^#y a-AcoEi^^ stasia y h^i^- 

[0 0 5 01 y 4 0 5 ©*tt^fitfiOi^ (± 

IS6) -l o) trffl^) , hv?-f^nfcmtz>7 

4 his— fr^x&mtit*^ h y 7 2 1 &*»U 
7) . 

[0051] m^, Vy'—f**^ 5/^a^*y5 
e°-:5fc-e&<5 (±159) 10) te«*fi) , B) #IB]$y 

f < . y h e-Tc-cfc 

a (±123) 4) 7) 8) ICffi^) , C) ZtlbyM 
(±12 2) 5) 6) left HO (D3m*)iCfr\?btlZ> 0 r 

ixtt. h*t&#y ^.-A^A) T*>*»g\ s®u 
b) x&zm-k. ^jjiy*— hatr— %\c*mmv>7 
[oo5 2] *i\ v4 bm^v =u-^mmmv^ 

[0 0 5 3] ^T7/S 4 0 8^M4 (±IBA) \Z 
<om&. ^%%y^ hr-^ii^^ h^yfa-^ 

y 7 2 i SrjE^b^ft^-^ <t ltjE^l*^^ 
o) o g}3tc^5v^rfi. h->— tr^x^mffim^^ 

hV72l_e. 7 2 1_f^^f5 0 

[00 54] Sfcfc, 7>f h^ya-A^SS!) 

11), ^7^7S4 1 i (±123) 4) 

7) s) io) infflyi) <a*§£\ hx-^tc^-r 



3?>f h^y^flffi«x> h y 7 2 1 «t«k^«i 
f-^iLt(EM*7>f 7 31C 

S^TS U7^7S4 12) 0 ®3iC53l^Ttt, 7^ 

F^y^f HtS^^M) 7 2 l_b, 7 2 1._ 

c, 721__d, 7 2 1.e*«at5o 

[0 0 5 5] ^7y7 p S 4 0 8(0*tt*4^ $ £ 

tT-TC-CfcS^JWf-*** U7^S413) o 
[0 0 5 6] ^ry-fS 4 1 3©*fr«4 (±I2B) 

Fr^^^^r-Y -y^^^^y 5 <0{R««— »K U it ± 

#Tf£ (^75/7 P S4 1 4) e EI3i;:*5l*Tti:* 7>T h 
^7^fliti^>'Mi7 2 1_b, 721_c, 
7 2 l_d*5fi^-t-5 0 ^ P>tC. ^yhTy^nfftol 
3 2^ftt5 (^y^S4 15) , ^ Ky^I 

1 1— « 

[0 0 5 7] yl^y^S 4 1 3©*fr^J*SC (±|SC) 

irta^) <rc^£\ SK7>f h^— *tt, H^JWy^e-h 
=f co^ ^cT-'L^ t°-7cT'^^^^*b, 7-f h7 

^/^S4 1 6) 0 dmi^Tte, ?<< bi'—tr>'X i B 

mmm^> b y i 2 i_a tm^-rzo 

^3^JE*f»B3 2$rHtT-t-S (^7^54 17), 
^7y7"S 4 1 8-^0 

hnb-^^k'-TcT'fcS^Mt (^7^541 
8) , ^7y^S4 18^M4^ (±IS2) 

4) e) s) i-fs^) . i^ssy^- h = t°-^=»^-5fe 
(^^^ h^-^^^-Y h^y^tm^y hy 7 

2 1 ^rt*Sr(EiH-f 5 (x^^/^S 4 19). ^ttl^XS 

y^—h^t^-^, 7^ h^^7^f hi^^f 

[0 0 5 9] ^C, ^!0^h7^^IC^«7^h 

*mmmwi7 3, ^<t^ jE*fb«-*9^ h^-^^^ 

U7y7°S 4 2.0) o -X^y^S 4 2 0 O^t^cil 
(±122) 5) 6) tdffiMO CO^, ^^7^f h7-i? 

ic*tJ&1-57-r hi/-^y^ei»*xy h y 7 2 Hi 
^■CifcSfcfc, 7>f h^y^flff^xy b y 7 

2 1 $r»»t5 U777°S421) o EI3JC*5V^T 

[ooeoliti;, ?4 h^7c(cy-r hffeT*n%- 

(^7^S4 04) U M^HTtSo 

[o o 6 1] ms^ &i&mmfx j rj*2<D^ ^mm 
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[oo62] mmttm?* hi/-fr>x<gmmn 
7 3ics»<**tfc7>r hs/-^y^fnt^x> h y 7 

2 l *»6(EiS»»«ra«i-5 ^S501). 

[0063] &c #n*y*- h = tf— 3fetw, s^l 
fc7>r h^-^v^sti&^v h y 7 2 i ir^-ra 

y^ h^-fZ^m-fZ {X"ry7°S 5 0 2) „ 
[0 0 6 4] jfctC, (E»Ufc9-f hV— 5r^«SflMR 

rc> h y 7 2 1 (om&tt&y'C b tr>-x<gmmm7 

3^<D&m*»f&1-Z> U7^S 5 0 3) . 

[0065] mc, m&Ltzyj ti/-tr>x<&mtftm 

4) o 777^5504^1^4^ ^mmv 
teaiLfc^ h->— ir^^^ait^vhy 7 2 1 

£#»L. 5 0 5). ^iHbfc7>r hr-^ 

as^ys/ajney 5cD{5^i#£{£/E Ltt^ci^l: 

mm&m&i>mtki~z> uf^s 506) 0 

[0 0 6 6] HJ6IC, fBttg«S<'*7 t A2£>rt, 

y^~ tr-oatr— jt#y a— A&#oia«SMa*> 
a 2 -entTi"S*PlffljE^:fi:ftia4 1 (:o^rp| 

[0067] sr, hf- trisxwmm 

[0 0 6 8] ftlC, »^LfcjE^k*l*©7>r by*-? 

^xiE^tm^mm-t^ (^5/7$ e 0 2) 0 

[0 0 6 9] ^f7^S 6 0 2^f«4^1^ 
S/^S 6 0 6^ o 

[0 0 7 0] 7f y^S 6 0 2(n0kfc^ff&iL<Dm&. IE 
(^7y7S 6 0 3) o 

[0 0 7 1] ttfcLtzy'f bis-tr^x^mm 
^xy h y 7 2 l (DlEjZitM&y^ bis—<tr>xVm#t 

&7 4^<D&m&Mfa1-Z> U7^S 604) 

t'7h-777 P I^!3 2^tT« U77/ 
S 6 0 5) o if y h^^^M0f^S3 2tCOt^-Cfi«Se 
i"6 0 ^^t, ^7-r/S 6 0 2— M60 
[0 0 7 2] iE^bUfc^-r bis—^rls^m^ 

m^-is b y 7 2 1 aHE&*t*7>f h ^>-*««t«R 

6) o ^7 7^S 6 0 6^«4^i^ #fa]#jy 



1 &A?#L (^r^^S 6 0 7). lESSfLLfc^-f 

i~<5 Ufy/S 6 0 8) 0 

[0 0 7 3] 17 1:, ffit§i£tE ^7^2(0^ »=>tr 
-5£#y V 1 £»olE«£«v'*:rA 2 -cswm- 
Stf > h^^^0#X.^»J^3 4tCo^r^"T 
60 y$tt*&»4, ^^7^7^8 10^ 
-Cl)§^7h-777 J 8 0 9 (2o^±#^E-fS^(-^ 
lS*MM69>f hv— ^v** 8 1 l^^rv^) coM#rr 

[0 0 7 4] Wl^mnL.tz\i^y b^y^WOWZ- 

j*mmz.mnisXi>&\,\ 

[0 0 7 5] £1\ ^=>f-5c^y ~— AV 1. 
t/, SR#y tf-5c#y ~-AV 1 

£t co * m = tr-* jj?y a -AV2t*«3 b°-5fe # y 

Z>tz#), ^7K77'§]t)fx.4'77^8 0 4^OFF 

x&zfrmm-rz (^777 i s7oi) 0 

[0 0 7 6] 777^S7 0 1^MM^ * 

tii if v b-?yy*m*)m7L*(ntz*b, M$r^7t5o 

[0 0 7 7] ^7 7^S7 0 1^M4O^ \fy 
h-7«7^^fx c l 3 77^8 0 4^ONt5 (Xy^yZf 
S 7 0 2) o 

ioo7 8] ftK, yfvb^v^mvmxsmtir*? 

^h^y7#^t, ffaflHR^y^tfy ^^ 
y:/jiO0#;t*«7*f h^^#8 0 6i:«l:St« 

(7777 8 S 7 0 3) 0 iryh^y^DtlS^att 

Da-^viH, ^ttsKy*— ASr«=fcr— Tc^y 
-*#y a -AV2t*«j3fcr-ftji?ya--i.v3oft 

mAstf?* bis- y-^XV&QLJ&L,, \fyb^y7°®«) 
mxMmy-i bis—fr % *sx# 8 06 t LTAt\> 

[0 0 7 9] ^fc, tf y b-ry-JmVWZMm?* bis 
— >7*>X# 8 0 6ri^ WVWZ-fttfy ^-7^8 0 9^ 

a-A©7^ h^y^^fStSi^ ^\^tz\f 

y b~* Wx-mmy-t bis->r^xn 8 o 6^ 

f Lt^#!) ^—J*<Dy 4 bf—f^^Xm*) %X 

•J#^R«7>f h^^#8 0 6^^^T^^4?y 



-8- 



-ittfy ~-j±v i t-?z>±x<Dtym^v— 9g#v ^— 

AV2ir«^t: 0 -^!)^-AV3(:i3^t, -SIC 
n^^-Y h^^#^, b^yzTS 

So 

[0 0 8 0] *Sf7>T hi/-^>*# 8 0 5**^ 

y h-ry^DSiSl7^ hi/— 8 06 tP 

-y>*#£ijj§:7y^8 0 8£rOF Fi"S (^^y^S 
7 04), 

[008 1] t'^h-7^7 p IW132l:tf7 

>r hv— 8 o 5 1 h-wywowJtmm? 

J h->— 8 0 6tf>Jttfc£rS#-t"*7>r h^>— 
^#?IBS*77^8 0 7S:ONt5 (^^S7 
0 5) o 

[0 0 8 2] RlC, V ^ — J*(D\M3g(D=* t°—9c3k 

TlcfcfU try Y-w?&*)**MMk5'{ bf—lrfx 

#80 6&<Ei£U *Sr^ hV- ^r^^ft^tTy 
y^«*l#iLJWl9>r h ^>-*# 8 0 6£ri&;ifc*§ 
fir ^ (^f^ 

S 7 0 6) 0 

[0 0 8 3] ^§£#y ^-Kcoy-i bis—fr^X 

SjMWtT* (^f^S 7 0 7) 0 XTy?*S 7 0 7 

8) . ffU^^y^S 7 0 7<Dmm&Mft-rZ>a ft*3* 

[0084] 7fy7°S 7 0 7^#jS4©i^, 
^ hco fc'y b-^yys 0 9c^F^j, le^^^^-f h v— fir 
V*# 8 1 1 ^M^^^y ^8 1 0£OFF U 

^tvhv^ysoQ^^yrti (^7^5 7 0 

9) 0 rftli, r^^-C, yiS*#y~-^£> y§8*# 
y a— A^iSS^ jc^y a-AV l £i-*£T(0*ra 
^ t°-3fe^y a-M2 ^"^V3 
<0^ffy^ h*>— -fir>-;*# 8 0 5 If? h-^y 

7*>-^#8 0 6£j*gx, St^^S' h 
-^y:/8 0 9^<D|B»SrM*&»^.Tfe*fc«>"e*)So 
[0 0 8 5] We, S»#!) a -A©|»03tf-^ 

S 7 1 0) o 

[0086] sB^ya-^ioasKoaf-fti 

(^y^S 7 1 1) o ^Xy^S 7 1 l<D**MFfS& 
tf>»-g\ — JfeB*|BWb Ury7S 7 12). Wl/XT 
y7"S7 1 1 0*l««T*tTi"*. 



[0 0 8 7) ^7y7 B S7 11^MM^ tfy 
77^8 0 4 ^OFFt Ufy^S 7 13), 
[0 0 8 8] @8IC % Efil^7^2^ *»!]=« 

^oiett^® ^fA2t' nfT-r s bf y h t y ?*w v m 

[0 0 8 9] tf, ^ f-jGlM' h^bt'7 
!)f^i7^h^>'^#8 0 6^U tfy h 

3£$y-f hi/- ^>*# 8 0 6(C|S:£1-6 C^yZ/S 
8 0 1) o 

[0 0 9 0] ^C, y-f h i/— ^ V^#iiJig^7^8 0 
8^0FFt U7^S 8 0 2) , y-Y 

^m^m^yyy's o 7^0Nt§ (*xy:/S8 0 

3) o 

[oo9i]^ att^y ^-Aiiii^^fc'-^ 
##i-£7HWrf£ U7>>ys 8 0 4) 0 

[0 0 9 2] *<ryZfS 8 0 4 (D&ft^F/dcSS: (Mja fc° 
-^ya-AV3Cffi^) £>4§£\ ^^y^S8 0 8 

[0 0 9 3] ^^y^S 8 0 4(D^#fiJc3t fcT— 

3t^y ^-AV2icti^) ^^^y^-AoiE 

lgtf>=ifc>— 5fc£-CtC*tU t*y Y^yzftyVmxMtiky 

4 his—fr^* #8 0 6 £$£i£U ^0f7-T h 

8 0 5^t'j/ b-W^WQWjLmm?* bis—v-^ 
* # 8 0 6 £jg3;L/d#£^y>r h ^ #PJS«^ 
Sr^fc-TS (*-ry:/S 8 0 5) . 

[0094] ^K#y3.-A<oar.»03tr-ft^ 
T^p>7^r hv— -firv^#i«a*^36s*5^*ij»rr5 

(^7^/S 8 0 6) o XTyZfS 8 0 6<D&{ttJ$.& 
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[0 0 9 5] ^ry7 P S 8 0 6^MM^ 
^^-A<D7^ hi/-fir^#iS77^0N-C* 
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<td^ x ~i&mf®m*> c^y^s 8i7), ^pu^^T- 
^/^s 8 i 6<Dmm&mn-tZo 

[0 10 3] ^xy^S 8 1 §<D$kfrf&<L<nm'&. 
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(*^y:/S 9 0 1) o 

[0 10 6] mmy-f h^>^# 8 0 5^:^ 
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y^ hV— <5r>^# 8 0 6 £|5]e, 
Wm-tZ (Xfy^S9 04) o 
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[0111] XTyZfS 9 0 4(^)«M^, 
hV— ^>-;*##iJii:7y ^8 0 8 £ON-T6 (^fy^ 
S 9 0 5) 0 

[0 112] fcl^ f h^y^Sfr^y ^8 1 0ri>O 
>/8 1 0£ONU iE^^7-r 8 1 1 
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S 9 0 6) e 
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gtc^bfc^fr^-^lr^^v^ 812^^!)^ 
>- h-f £ (^r^s 9 0 7) o 

[0 114] h^-^^#MI^77 
;/8 0 7£OFFL (^7^890 8) , ^H^fct-T 
-f So 

[0115] iioii, f y h^y7W*)Wz.\z.teii 
[0116] B=>e-7t7$v ~-j±v i $r^o-7--r h-e 

HfTbTV^O^tf y h'Vy^ , 9#X.®tb^3®3 4, 
^P^fc°— ft# , J:*-.AV2, *fcfi, ^iss^^-ife^ 
y a-AV3^ot^ hVm'<Tl'X\,^Z><D&¥y Y~? 
^7 p ^3!)fx:M3 5r'$)6o fcAcU -St'yh^y 

[0 117] ilOfi, t>f h£#c£>fcfy h^y^l^ 
lb fcf y ^7^2^^^ hvj/^^fx.^il^ 
LT*^, fc'y h^y^U9#X-te. ^<D'&, Vyh-? 
y7 a 2^6^5/h'7^^1^ ££>l-> tTyh^y^l 
a»6 ify h^y;/2 — iri^ofcJ: 54-, ^Sic^tri - 

[0118] try v^yyowmx.*, ^ Y±ftb b 

try h v 7 7 6 ^llft s^i-^Tt 5 r <b (c i ^ tr 
m±X'^-rtmmithy4 i l&mc* 

[0 119] tTy h-vy^§J9#X.(C*JV^T, - 

tP<0^-f KoPWW(WJlfc»*, tfyb^y^»9» 

x^m^^r hv— 8 o 6<D&°y-'< Y<D^m%tW 

*5l^r> ^7y7'S7 0 9Oe'y h-^y^8 0 9^!)T 
Op^^^HtT^-fx 7^S7 10CfcV^t, tf y h 

[0 12 0] ^^(C. #|Ei»»Bi/X7 i A2*Ci3l>Ttr 
7hvy7 P 80 9$:3l^±^ — »^ K^P*#«r 

fflLXV^tTy h'7y7 , 809^^ U T L^V N <t 9 1^ 
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[0122] 11 ltc, h&1&&s *M MB"CsJ?y 
[0 12 3] £?\ f-^ hft&KJ: «Jii:»^3^-7c«r 
atf'J rL-Airl^CS^^^-TC^^ a— 1 &#o#y 

8 0 5&MMf&&y » — A^il^-fS (^f^S 
110 1) o 

[0 12 4] W:, h ^/c^ii:^^ 

y-y-fS 1 1 0 2) o 
[0 12 5] ^ Ytmttfc&to^V—tt 

6 Ufy^S 1 1 0 3) o r*U^P$. a 

So 

[0 12 6] ffC, gi^-f Yb%\tLt£WM<D-=m— 7Z 
(XTyfS 110 4). 

[oi2 7] i»3^-tti3 6iifcT»t>, -to**y 

[0 12 8] ifEOJfffcfcKSSoa tr-TcU-'f b£ L 

lo 1 2 9] hi -^-r h«8i«. b«tf#y 

[0130] £i\ mttW&<o=i hlZlo^ 

a*^ H4:»ffc*aa*= i t! , '-5c^ hco^*^ h-c 

^8 0 9^ r<fc*f*5rfl&fci~fcrs' h-^y^WB 

1 j&s«rLV^<E>^?/ 0 9 SrillRU tf y h-v 

^-^^^-rSfcfecoH^f y h^ y^8 l 3l:^t° 
— Wy^S 12 0 1) o mft^y b^y^8 1 

i*, -y->r h«»«t>y*-h = tr-*s»«urt^»& 
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vis***}) <0B!«<&— flJSr^"*"* 
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-7c # y a - a S:»ois«KiR f * ^ a -c^r a tr y 

[19] *IBWo9IJSo«l»^*3^S, b^S, ^5 
S<^—^J$r^'t-o 
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[112] *»W<©*lfeo^(Btc*3^S, Mft^ 

a. b^-c^y Art**:— ***«fc*«>*w 
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